DiscoverHow AI Is BuiltData Processing for AI, Integrating AI into Data Pipelines, Spark | ep 16
Data Processing for AI, Integrating AI into Data Pipelines, Spark | ep 16

Data Processing for AI, Integrating AI into Data Pipelines, Spark | ep 16

Update: 2024-07-12
Share

Description

This episode of "How AI Is Built" is all about data processing for AI. Abhishek Choudhary and Nicolay discuss Spark and alternatives to process data so it is AI-ready.

Spark is a distributed system that allows for fast data processing by utilizing memory. It uses a dataframe representation "RDD" to simplify data processing.

When should you use Spark to process your data for your AI Systems?

→ Use Spark when:

  • Your data exceeds terabytes in volume
  • You expect unpredictable data growth
  • Your pipeline involves multiple complex operations
  • You already have a Spark cluster (e.g., Databricks)
  • Your team has strong Spark expertise
  • You need distributed computing for performance
  • Budget allows for Spark infrastructure costs

→ Consider alternatives when:

  • Dealing with datasets under 1TB
  • In early stages of AI development
  • Budget constraints limit infrastructure spending
  • Simpler tools like Pandas or DuckDB suffice

Spark isn't always necessary. Evaluate your specific needs and resources before committing to a Spark-based solution for AI data processing.

In today’s episode of How AI Is Built, Abhishek and I discuss data processing:

  • When to use Spark vs. alternatives for data processing
  • Key components of Spark: RDDs, DataFrames, and SQL
  • Integrating AI into data pipelines
  • Challenges with LLM latency and consistency
  • Data storage strategies for AI workloads
  • Orchestration tools for data pipelines
  • Tips for making LLMs more reliable in production

Abhishek Choudhary:

Nicolay Gerold:

Comments 
Top Podcasts
The Best New Comedy Podcast Right Now – June 2024The Best News Podcast Right Now – June 2024The Best New Business Podcast Right Now – June 2024The Best New Sports Podcast Right Now – June 2024The Best New True Crime Podcast Right Now – June 2024The Best New Joe Rogan Experience Podcast Right Now – June 20The Best New Dan Bongino Show Podcast Right Now – June 20The Best New Mark Levin Podcast – June 2024
In Channel
Limits of Embeddings: Out-of-Domain Data, Long Context, Finetuning (and How We're Fixing It) | S2 E5

Limits of Embeddings: Out-of-Domain Data, Long Context, Finetuning (and How We're Fixing It) | S2 E5

2024-09-1946:06

RAG at Scale: The problems you will encounter and how to prevent (or fix) them | S2 E4

RAG at Scale: The problems you will encounter and how to prevent (or fix) them | S2 E4

2024-09-1250:09

From Keywords to AI (to GAR): The Evolution of Search, Finding Search Signals | S2 E3

From Keywords to AI (to GAR): The Evolution of Search, Finding Search Signals | S2 E3

2024-09-0552:16

Data-driven Search Optimization, Analysing Relevance | S2 E2

Data-driven Search Optimization, Analysing Relevance | S2 E2

2024-08-3051:14

Query Understanding: Doing The Work Before The Query Hits The Database | S2 E1

Query Understanding: Doing The Work Before The Query Hits The Database | S2 E1

2024-08-1553:02

Season 2 Trailer: Mastering Search

Season 2 Trailer: Mastering Search

2024-08-0804:16

Unlocking Value from Unstructured Data, Real-World Applications of Generative AI | ep 17

Unlocking Value from Unstructured Data, Real-World Applications of Generative AI | ep 17

2024-07-1636:28

Data Processing for AI, Integrating AI into Data Pipelines, Spark | ep 16

Data Processing for AI, Integrating AI into Data Pipelines, Spark | ep 16

2024-07-1246:26

Building AI Agents for the Enterprise: Realistic Use Cases, Cost Controls, Seamless UX | ep 15

Building AI Agents for the Enterprise: Realistic Use Cases, Cost Controls, Seamless UX | ep 15

2024-07-0435:12

Building Predictable Agents: Prompting, Compression, and Memory Strategies | ep 14

Building Predictable Agents: Prompting, Compression, and Memory Strategies | ep 14

2024-06-2732:14

Data Integration and Ingestion for AI & LLMs, Architecting Data Flows | changelog 3

Data Integration and Ingestion for AI & LLMs, Architecting Data Flows | changelog 3

2024-06-2514:53

ETL for LLMs, Integrating and Normalizing Unstructured Data | ep 13

ETL for LLMs, Integrating and Normalizing Unstructured Data | ep 13

2024-06-1936:48

Serverless Data Orchestration, AI in the Data Stack, AI Pipelines | ep 12

Serverless Data Orchestration, AI in the Data Stack, AI Pipelines | ep 12

2024-06-1428:06

Mastering Vector Databases: Product & Binary Quantization, Multi-Vector Search

Mastering Vector Databases: Product & Binary Quantization, Multi-Vector Search

2024-06-0740:06

Building Robust AI and Data Systems, Data Architecture, Data Quality, Data Storage | ep 10

Building Robust AI and Data Systems, Data Architecture, Data Quality, Data Storage | ep 10

2024-05-3145:33

Modern Data Infrastructure for Analytics and AI, Lakehouses, Open Source Data Stack | ep 9

Modern Data Infrastructure for Analytics and AI, Lakehouses, Open Source Data Stack | ep 9

2024-05-2427:53

Knowledge Graphs for Better RAG, Virtual Entities, Hybrid Data Models | ep 8

Knowledge Graphs for Better RAG, Virtual Entities, Hybrid Data Models | ep 8

2024-05-2036:40

Navigating the Modern Data Stack, Choosing the Right OSS Tools, From Problem to Requirements to Architecture | ep 7

Navigating the Modern Data Stack, Choosing the Right OSS Tools, From Problem to Requirements to Architecture | ep 7

2024-05-1738:12

Data Orchestration Tools: Choosing the right one for your needs | ep 6

Data Orchestration Tools: Choosing the right one for your needs | ep 6

2024-05-1032:37

Building Reliable LLM Applications, Production-Ready RAG, Data-Driven Evals | ep 5

Building Reliable LLM Applications, Production-Ready RAG, Data-Driven Evals | ep 5

2024-05-0329:40

loading
Google Play
Download from Google Play
Castbox
Download from App Store
usUnited States
00:00
00:00
x

0.5x

0.8x

1.0x

1.25x

1.5x

2.0x

3.0x

Sleep Timer

Off

End of Episode

5 Minutes

10 Minutes

15 Minutes

30 Minutes

45 Minutes

60 Minutes

120 Minutes

Data Processing for AI, Integrating AI into Data Pipelines, Spark | ep 16

Data Processing for AI, Integrating AI into Data Pipelines, Spark | ep 16

Nicolay Gerold